The optical fiber transmission system of low cost and high capacity is the development direction in the future optical communication. In order to further reduce the cost of single bit transportation and improve the capacity of single optical fiber, using the advanced modulation format having more compact spectrum is a good solution. The Partial Response Quadrature Amplitude Modulation (PR-QAM) is also called as Quadrature Partial Response (QPR) system, as it has a high spectral efficiency, related studies have been carried out in the field of optical communication. PR-QAM is a modulation format that combines the characteristics of the partial response and the quadrature amplitude modulation. The partial response system achieves a high spectral efficiency by introducing controlled Inter-Symbol Interference (ISI). In the partial response system, the first category of partial response signal is also called as duobinary signal, which has a smooth spectrum and can be generated by a physically implementable filter, and hence is widely studied and applied.
Currently, the coherent receiver device provided with respect to the PR-QAM optical communication system does not adopt adaptive equalization technique. In order to exert the advantages of the digital signal processing (DSP) technique in functions and performances, the adaptive equalization technique is expected to be widely used in the coherent receiver of the PR-QAM system to overcome the linear damage caused by the link. However, the constant modulus algorithm (CMA) or its improved equalization algorithm commonly used in the coherent receiver cannot be directly used in the PR-QAM system. This is because an important precondition of the CMA and its improved algorithm is that the signal to be processed shall meet the statistic characteristic requirement of independent identical distribution, while the PR-QAM system violates the requirement because it introduces the controlled ISI, thus the receiver device needs to be redesigned. The method of inserting digital pre-filter and post-filter in front of and behind the CMA-based adaptive equalization module respectively can effectively solve this problem. However, the conventional technique assumes that the system does not have the problem of carrier phase recovery in the typical optical coherent communication system, or has eliminated the problem of phase mismatching by using the phase lock loop technique of the receiver front end. But due to the defect caused by the loop-delay of the phase lock loop, the equalization is generally expected to be performed in the optical communication system prior to the phase recovery in the digital domain. In addition, the pre-filter device in the conventional technique is relatively complicated, which increases the complexity of hardware.
Literatures beneficial for understanding the present invention and the prior art are listed as follows, and they are incorporated herein by reference, as if completely described in this text.    1. “Partial response, quadrature amplitude modulation system” Tadayoshi Katoh, U.S. Pat. No. 4,055,727, Oct. 25, 1977.    2. “Method and apparatus for partial response demodulation”, Leo Montreuil, U.S. Pat. No. 5,214,390, May 25, 1993.    3. Kazuro KIKUCHI, Yuta ISHIKAWA, and Kazuhiro KATOH, “Coherent Demodulation of Optical Quadrature Duobinary Signal with Spectral Efficiency of 4 bit/s/Hz per Polarization,” ECOC 07, Sep. 16-20, 2007 Berlin, Germany.    4. Ilya Lyubomirsky, “Quadrature Duobinary for High-Spectral Efficiency 100G Transmission,” Journal of Lightwave Technology, to be published (www.ee.ucr.edu/˜ilyubomi/JLT-11759-2009-Final.pdf)    5. I. Lyubomirsky, “Quadrature duobinary modulation for 100G transmission beyond the Nyquist limit,” to be presented in Optical Fiber Communication Conference (OFC), paper OThM4, San Diego, USA, March 2010.    6. Peter Kabal and Subbarayan Pasupathy, “Partial-response signaling,” IEEE Transactions on Communications, Vol. 23, No. 9, pp. 921-934, September 1975    7. Jitendra K. Tugnait and Uma Gummadavelli, “Blind Equalization and Channel Estimation with Partial Response Input Signals,” IEEE Transactions on Communications, vol. 45, no. 9, pp. 1025-1031, September 1997    8. Ezra Ip and Joseph M. Kahn, “Feed forward Carrier Recovery for Coherent Optical Communications,” Journal of Lightware Technology, vol. 25, no. 9, pp. 2675-2692, September 2007    9. Andreas Leven, Noriaki Kaneda, Ut-VaKoc, and Young-Kai Chen, “Frequency Estimation in Intradyne Reception,” IEEE Photonic Technology Letters, vol. 19, no. 6, pp. 366-368, March 2007.